Inter-transcriber reliability of toBI prosodic labeling

The goal of this study was to evaluate the reliability among transcribers of a standard prosodic labeling system under relatively optimal conditions of training, supervision, facilities, procedures, and extent of speaker familiarity. The ToBI (Tones and Break Indices) model for standard American English[7][1] was used in the study; break indices indicate the degree of junction between words, pitch accents designate word prominence, and edge tones mark phrase boundaries. The American English speech corpora were read by a female professional speaker and by a male professional speaker, and were composed of several types of texts to ensure prosodic variety. Each of four experienced transcribers independently labeled each corpus. For each corpus, word level agreement in break indices, pitch accents, and edge tones between all possible pairs of transcribers was analyzed, and various statistics were calculated. Agreement among labelers was generally higher than that reported in previous studies[6][3] of larger and more diverse groups of labelers. Agreement was high for some prosodic categories, but low for others. The extent of reliability for various prosodic distinctions has important implications for re ning the ToBI model and for limitations in the use of prosody in speech technologies.